Learning to Teach Reinforcement Learning Agents

In this article we study the transfer learning model of action advice under a budget. We focus on reinforcement learning teachers providing action advice to heterogeneous students playing the game of Pac-Man under a limited advice budget. First, we examine several critical factors affecting advice quality in this setting, such as the average performance of the teacher, its variance and the importance of reward discounting in advising. The experiments show the non-trivial importance of the coefficient of variation (CV) as a statistic for choosing policies that generate advice. The CV statistic relates variance to the corresponding mean. Second, the article studies policy learning for distributing advice under a budget. Whereas most methods in the relevant literature rely on heuristics for advice distribution we formulate the problem as a learning one and propose a novel RL algorithm capable of learning when to advise, adapting to the student and the task at hand. Furthermore, we argue that learning to advise under a budget is an instance of a more generic learning problem: Constrained Exploitation Reinforcement Learning

Microbial diversity in the thermal springs within Hot Springs National Park

The thermal water systems of Hot Springs National Park (HSNP) in Hot Springs, Arkansas exist in relative isolation from other North American thermal systems. The HSNP waters could therefore serve as a unique center of thermophilic microbial biodiversity. However, these springs remain largely unexplored using culture-independent next generation sequencing techniques to classify species of thermophilic organisms. Additionally, HSNP has been the focus of anthropogenic development, capping and diverting the springs for use in recreational bathhouse facilities. Human modification of these springs may have impacted the structure of these bacterial communities compared to springs left in a relative natural state. The goal of this study was to compare the community structure in two capped springs and two uncapped springs in HSNP, as well as broadly survey the microbial diversity of the springs. We used Illumina 16S rRNA sequencing of water samples from each spring, the QIIME workflow for sequence analysis, and generated measures of genera and phyla richness, diversity, and evenness. In total, over 700 genera were detected and most individual samples had more than 100 genera. There were also several uncharacterized sequences that could not be placed in known taxa, indicating the sampled springs contain undescribed bacteria. There was great variation both between sites and within samples, so no significant differences were detected in community structure between sites. Our results suggest that these springs, regardless of their human modification, contain a considerable amount of biodiversity, some of it potentially unique to the study site